Sampling Techniques for Zero-sum, Discounted Markov Games

In this paper, we first present a key approximation result for zero-sum, discounted Markov games, providing bounds on the state-wise loss and the loss in the sup norm resulting from using approximate Q-functions. Then we extend the policy rollout technique for MDPs to Markov games. Using our key approximation result, we prove that, under certain conditions, the rollout technique gives rise to a policy that is closer to the Nash equilibrium than the base policy. We also use our key result to provide an alternative analysis of a second sampling approach to Markov games known as sparse sampling. Our analysis implies the (already known) result that, under certain conditions, the policy generated by the sparse-sampling algorithm is close to the Nash equilibrium. We prove that the amount of sampling that guarantees these results is independent of the state-space–size of the Markov game.